Research of user behaviour in food sales mobile app

Table of Contents

Data preparation

We prepared the dataset for the further analysis.

Data check

We have data for the period from July 25 to August 7, 2019.

We see a very small number of observations for the period from 07/25/2019 till 08/01/2019. We know that technically for some users events from the past can be recorded in the logs in the upcoming days, which causes skew data. The histogram shows a noticeable increase in observations for the period 07/31/2019 till 08/01/2019. Let's look at the histogram by the time for the events on July 31.

We see a significant increase in number of observations at 9 p.m. We can remove all the data before this time. As a result, we'll have the data beginning from 9 p.m. 07/01/2019 till 08/07/2019.

With this truncation, we lost a negligible percentage of the original data. Let's check that we have users from all three experimental groups.

Funnel

There are five type of events in the logs, the rearest is an appeal to the tutorial, most often - the appearance of the main screen. In the Offer-Cart-Purchase subsequence, the number of users decreases.

Apparently, the events are arranged in the following order:

  1. Main screen (MainScreenAppear)
  2. Offer Screen (OffersScreenAppear)
  3. Cart Cart Screen (CartScreenAppear)
  4. Successful payment confirmation screen (PaymentScreenSuccessful).

It is unclear at what stage there is an appeal to the tutorial. We make an assumption this event can be ignored for further analysis.

The maximum user lost occurs at the second step of the funnel (OffersScreenAppear).

Observing the experiment results

The discrepancy in numbers between the groups is about 1%. There are 2 control groups for the A/A experiment - 246 and 247. Let's check if there is a difference between samples 246 and 247 (the hypothesis of equality of shares).

246+247 - combined control groups.

The most popular event is MainScreenAppear. Let's count the number of users who committed this event in each of the control groups.

More than 98% of users take this action in each of the groups.

H0 - there are no differences between the shares, H1 - there are differences between the shares.

Test for groups 246, 247

H0 - there are no differences between the shares, H1 - there are differences between the shares.

There is no statistically significant difference between the shares, the division into groups was correct. 248 - the group with the changed font. We will conduct a similar test for her.

Test for groups 246, 248

H0 - there are no differences between the shares, H1 - there are differences between the shares.

Test for groups 247, 248

H0 - there are no differences between the shares, H1 - there are differences between the shares.

Let's compare the results with the combined control group.

Test for groups 246+247 (control), 248

H0 - there are no differences between the shares, H1 - there are differences between the shares.

Conclusion

We made 16 tests of statistical hypotheses using a significance level of 1% for control groups and a significance level of 5% for the rest.

The tests did not show a statistically significant difference between the groups - changing fonts in the application does not scare away users on their way through the funnel.

We lose the most users at the second step of the funnel - OffersScreenAppear, it's probably worth considering how to increase user interaction at this step.